Add plugin registry and callbacks for AI model validation #26309
+375
−15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stefanberger
requested review from
jeejeelee,
WoosukKwon,
robertgshaw2-redhat,
njhill,
ywang96,
comaniac,
alexm-redhat,
aarnphm,
chaunceyjiang and
22quinn
as code owners
|
@njhill FYI |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a plugin registry for AI model validation, allowing for integrity and provenance checks on models and LoRA adapters. Callbacks are added at various points in the model and adapter loading process to invoke these validation plugins. The changes span across model loaders, LoRA management, and the V1 engine core, with corresponding tests to ensure the validation mechanism is triggered correctly. The error reporting for LoRA loading failures is also improved. My review found a critical issue in the GGUF model loader that would prevent loading models from a URL.
Add a model validation plugin registry where classes implementing the ModelValidationPlugin interface can be registered. Enable the validating on local models that have already been downloaded by the user. Add a test case with an already downloaded model whose config.json is unmodified so that a ModelConfig can be created from it. Signed-off-by: Stefan Berger <[email protected]>
Extend a LoRARequest with a validate() method to enable validation of a LoRA adapter when it is loaded. Add a test case. Signed-off-by: Stefan Berger <[email protected]>
Implement a method 'validate' in the BaseModelLoader that first checks whether any plugin requests to validate the given model and then possibly downloads all the model files, including the signature. For this, query the subclass of BaseModelLoader for its download type. Support validation of local models and those downloaded from Huggingface Hub. Add a test case. Signed-off-by: Stefan Berger <[email protected]>
Extend the reporting of an error over RPC by the cause of the error if it is known. This then for example not only reports that the signature verification failed but also the reason, such as when an unsigned file was found. Signed-off-by: Stefan Berger <[email protected]>
…aded The reason why a LoRA adapter could not be loaded may include information from model validation, such as that model signature verification did not succeed because unsigned files were found. Signed-off-by: Stefan Berger <[email protected]>
stefanberger
force-pushed
the
validation-plugin.4upstream
branch
from
0087ecd to
a150791
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
This PR adds a plugin registry for AI model validation plugins and sets callbacks from which the plugins are invoked. The model validation can be used on AI models and LoRA adapters and therefore the plugin points are set to verify:
The first plugin to use this new infrastructure will be used for integrity and provenance verification of AI models and LoRA adapters and will be hosted outside the vLLM repository.
Test Plan
The following new tests have been added:
The following existing test have been extended:
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.